1. Field of the Invention
The present invention relates to a multimedia playing apparatus for playing multimedia information items (referred to in the following as multimedia titles), each consisting of a combination of multimedia data objects such as video clips, audio clips, still pictures, text items, etc., with playing conditions (e.g., playing intervals, window size and window position on a display screen, etc) for the various multimedia data objects being specified in a scenario.
2. Description of the Prior Art
When playing a multimedia title, it is necessary that the start and end of playing the various monomedia objects, during the playing progression, be linked to some reference timing axis, in a manner that is specified in the scenario data. With a prior art type of multimedia playing apparatus, a real-time axis is generally used for that purpose. For example, the scenario of a multimedia title might specify that a certain still-picture monomedia object is to start to be displayed at a point which occurs 3 minutes after playing of that multimedia title has started, and is to cease to be displayed at a point which occurs 6 minutes after playing of the multimedia title has started. That is to say, playing times for the various monomedia objects of a multimedia title, specified in the scenario, are linked to points along a real-time axis whose origin is the time point at which playing of the multimedia title begins.
However in practice, such a method presents various basic problems. For example, the multimedia title might include a video object, i.e., a video clip, which is played for a relatively long duration during the playing progression of the multimedia title, and it might be required for a certain still picture to start to be displayed at a certain point (such a point being referred to in the following as a processing time point), within that progression, which is accurately determined in relation to playing the video object. In a practical type of multimedia playing apparatus, in which the data of the video object might be transferred from a data storage device via a network data communication path before being displayed, it cannot be guaranteed that the progression of playing that video object will be accurately in accordance with a predetermined absolute real-time axis. There will be minor variations in timing, which will not affect the playing of the video object as seen by a user, but whereby it cannot be ensured that a certain frame of the video will occur at an accurately predetermined point on the real-time axis, after the start of playing the multimedia title. Moreover, the data of a finite-time object such as a video object are generally stored and transferred encoded in a compressed format such as the MPEG1 format. When playing of such a video object is started the data are successively decompressed (decoded) by a decoder apparatus, in a manner which is not linked to the progression of the scenario. Thus, only the starting point of playing such a finite-time object is directly linked to the playing progression of the multimedia title as set out in the scenario.
The longer the duration for which such a finite-time object is played, the greater will be the amount of deviation between the actual time of occurrence of a certain part of the object (e.g., playing of a specific frame of a video object) and the processing time point which has been specified for that occurrence in the scenario. As a result, there may be significant errors between the scenario and the multimedia title xe2x80x9cstoryxe2x80x9d which is actually presented, e.g., still-picture objects may not presented at the correct times during playing of a video object. Thus the problem arises that the xe2x80x9cstoryxe2x80x9d which is actually presented by a multimedia title may vary in accordance with the operating environment, i.e., in accordance with the characteristics of data storage devices, a network through which data are transferred, etc.
It can thus be understood that the above problem results from the fact that management of playing of the multimedia title is based upon use of a real-time axis as a reference axis for the playing progression.
A second problem is as follows. It can be assumed in general that some finite-time object, and in particular a video object, will be being played at any particular point during the progression of a multimedia title. When a monomedia object such as a still picture is to be presented, the data representing that monomedia object are read out from data storage (in general, as a data file), are converted to suitable form for being played, i.e., are formed in accordance with the scenario contents, such that the monomedia object will appear at the specified position, with the specified window size, on the display screen. However in a practical apparatus, such processing, and also the processing whereby the video object data are processed must be performed by the same computer, i.e., the same CPU. Hence, the acquisition of the still-picture object data and the conversion of these data to suitable form for presentation will impose a substantial load on that CPU, i.e., will require a significant amount of processing time. This may obstruct the processing which must be performed, at regular intervals, to acquire and process video data to play successive frames of the video data, and so may result in momentary interruptions of playing a finite-time object such as a video object.
A third problem is as follows. With a prior art type of multimedia playing apparatus, playing of a multimedia title is performed as an automatic sequence, i.e., the multimedia title must be played continuously in accordance with the scenario, from the beginning. It has not been possible to provide a function whereby a user can execute manual control of the playing progression, e.g., whereby the user can move the playing position to an arbitrary point within the playing progression. The reason for this is as follows. With the prior art type of multimedia playing apparatus, processing points (i.e., points at which playing of a monomedia object is to be started or terminated) are defined within the scenario at positions along the playing progression axis of the multimedia title, as respective time points. In order to be able to move the playing position to an arbitrary point along the playing progression, it is necessary first to determine the status of playing at that arbitrary point, and to obtain that information it would be necessary to again begin to read through the scenario, from the start of the scenario, until the specified time point is reached, and then obtain information specifying the monomedia object or monomedia objects to be presented at that time point. However if such processing were to be executed during actual playing of a multimedia title, it would be difficult to achieve a sufficiently rapid and smoooth transition to the new playing position.
A fourth problem is as follows. With a prior art type of multimedia playing apparatus, it has not been possible for a user to execute an operation to alter any of the display attributes of a monomedia object such as displayed (window) position, displayed (window) size, etc., i.e., to execute any operation (referred to in the following as a user-mediated operation) which results in a change in the progression of the multimedia title which is not in accordance with the scenario.
A fifth problem is as follows. With a prior art type of multimedia playing apparatus, it is not possible to execute any processing other than playing of multimedia titles. If it is necessary to execute other types of processing, then it becomes necessary to provide a dedicated apparatus for that purpose. If it were possible for the multimedia playing apparatus to execute external applications (for example, starting at any arbitrary point during playing of a multimedia title, as specified by an input command generated by a user while viewing the multimedia title), then various additional functions could easily be provided by such a multimedia playing apparatus, and the range of use would be greatly expanded.
It is an objective of the present invention to provide a multimedia playing apparatus for playing a hypermedia title, whereby the problems of the prior art set out hereinabove can be overcome. The term xe2x80x9chypermedia titlexe2x80x9d as used herein signifies a multimedia title which includes a finite-time monomedia object such as a video object which is played continuously from start to end of the playing progression of the multimedia title, with that finite-time monomedia object being formatted as a successive of playing sections which are of identical length. Such a video object will be referred to in the following as the base-axis object of the hypermedia title. For example, if the base-axis object is a video object (i.e., a video clip which is played continuously from start to end of playing the multimedia title), then that is inherently divided into a succession of playing sections, i.e., the video frames. Alternatively, an audio base-axis object could be used, for example by embedding periodic timing markers in an audio sound track of a multimedia title, to thereby define successive playing sections. With the present invention, the aforementioned processing time points are defined as respective numbers of these playing sections, so that respective attainments of the processing time points during the playing progression of the hypermedia title can be detected by maintaining a count of the number of playing sections which have been played, by a counter which will be referred to as the playing progression counter.
It should be noted that the term xe2x80x9cplayedxe2x80x9d as used in the following description and in the claims is to be understood in a very broad sense, i.e., to apply to playing of a finite-time monomedia object such as a video or audio object to display a moving picture or generate audible sound, and to apply also to static displaying of monomedia objects such as still pictures or text items.
As a result of defining the processing time points as described above, it becomes possible to accurately synchronize the various processing operations applied to the other monomedia objects of the hypermedia title (these being referred to in the following as the non base-axis objects) with the playing progression of the base-axis object, since these time points are linked to that playing progression, rather than to points along the real-time axis.
Thus,the present invention provides a multimedia playing apparatus which utilizes a specific finite-time object of a multimedia title to define a reference timing axis, which will be referred to as the base axis, and which is utilized (rather than using a real-time axis for timing reference purposes) during playing of a multimedia title to define the timings of respective processing points, i.e., the points at which playing of other monomedia objects is started or terminated, as defined in the scenario.
Hence, an apparatus according to the present invention inherently overcomes the first of the problems set out hereinabove.
To overcome the second problem set out above, the invention provides a multimedia playing apparatus whereby, prior to the time at which playing of a hypermedia title is to be enabled (i.e., prior to the time at which the apparatus is set in a condition in which it is possible for a user to input a command to specify that the hypermedia title is to be played), the data for all of the monomedia objects other than the base-axis object are acquired (e.g., read out as respective data files from a data storage device) and formed into corresponding data which are ready to be supplied to a playing device (e.g., video display section) for immediate presentation, with these data then being stored locally in memory until required during playing of the hypermedia title. As a result, during playing of the hypermedia title, each base-axis object can immediately start to be presented at the point which is specified in the scenario, without danger of obstructing the playing of the base-axis object.
To overcome the third problem set out above, the invention provides a multimedia playing apparatus whereby a hypermedia title is treated as a succession of segments, and whereby specific data (object presentation status management data) are prepared prior to the time at which playing of the hypermedia title is to be enabled, with the data specifying the processing which is to be executed within each of the segments. That is to say, for each segment, the corresponding part of the object presentation status management data specifies the monomedia objects (other than the base-axis object) which are to start to be played at the beginning of that segment, or for which playing is to be terminated at the beginning of that segment, or which are to continue (i.e., from the preceding segment) to be played during that segment. The object presentation status management data are stored in memory. Subsequently, during playing of the hypermedia title, if it is required to move the playing position to a different position within the playing progression, the necessary information concerning the playing status at that different position is immediately available, by reading out from memory the part of the object presentation status management data which corresponds to that new position. In that way, it becomes possible to provide a manual control feature, whereby a user can arbitrarily shift the playing position to a different position within the playing progression of a hypermedia title.
To overcome the fourth problem set out above, the invention provides a multimedia playing apparatus whereby, for one or more monomedia objects of a hypermedia title, data are prepared which specify at least one different playing attribute which is different from the attributes which are specified for that monomedia object in the scenario. For example in the case of a still-picture object, data may be prepared which specify a larger display size, and/or a different display position, from the size and/or position specified in the scenario. Such data are stored in memory prior to the time at which playing of a hypermedia title is to be enabled, and can subsequently be used to enable interactive control of that playing attribute (or attributes) by the user. Such a method of interactive control requires only a very small amount of processing, and so can be easily implemented during playing of a hypermedia title without the danger of obstructing the progression of playing the hypermedia title.
To overcome the fifth problem set out above, the invention provides a multimedia playing apparatus whereby information required for activating specific external applications, which are unrelated to playing of a hypermedia title, is contained in the scenario, and whereby a user can input an external application activation command, during playing of the hypermedia title, to selectively specify execution of one of the external applications.
More specifically, according to a first aspect the invention provides a multimedia playing apparatus for playing a hypermedia title formed of a base-axis object and at least one non base-axis object, the base-axis object being a finite-time monomedia object having a playing duration which extends from start to end of a playing progression of the hypermedia title, the base-axis object being configured as a plurality of sequential fixed-length playing sections, the multimedia playing apparatus comprising:
scenario data storage means for storing data of a scenario of the hypermedia title, the scenario including a plurality of sets of synchronization trigger data, each of the sets identifying a corresponding one of the non base-axis objects and specifying, for the corresponding non base-axis object, a processing time point during the playing progression at which a processing operation for starting playing of the non base-axis object or a processing operation for ending playing of the non base-axis object is to be executed, each of the processing time points being expressed as an integral number of the playing sections,
subject matter data storage means for storing respective subject matter data of the base-axis object and each of the non base-axis objects;
timer event generating means for periodically generating timer events;
base-axis object display means controllable for acquiring subject matter data of the base-axis object from the subject matter data storage means and utilizing the subject matter data to play the base-axis object, and for counting successively played ones of the playing sections;
non base-axis object display means for playing each of the non base-axis objects;
internal data memory means;
processing event counter means;
playing progression counter means for counting successive ones of the playing sections during playing of the hypermedia title;
playing command input means, operable by a user for generating at least a xe2x80x9cplayxe2x80x9d input command to designate that playing of the hypermedia title is to be started; and
overall control and synchronization processing means for
executing initial processing to set the apparatus in a condition of readiness for playing the hypermedia title, the initial processing comprising at least operations for acquiring the scenario data from the scenario data storage means, obtaining the sets of synchronization trigger data from the scenario data and storing the sets in sequentially numbered locations in the internal memory means,
for setting the processing event counter means and playing progression counter means to a common initial value, and
on completion of the initial processing, when input of the xe2x80x9cplayxe2x80x9d input command occurs, executing operations for
controlling the base-axis object display means to begin to acquire the base-axis object subject matter data,
in response to occurrence of each of the timer events, obtaining respective current values of the playing sections count and processing event count, obtaining from the internal memory means a set of the synchronization trigger data having a storage location which corresponds to the processing event count value, judging whether the playing sections count value is greater than or equal to a processing time point value which is specified in the obtained set of synchronization trigger data, and, when the playing sections count value is judged to be greater than or equal to the specified processing time point value,
executing a processing operation which is specified in the synchronization trigger data set, for a non base-axis object which is identified in the synchronization trigger data set, and
incrementing the processing event counter means.
According to a second aspect, with such a multimedia playing apparatus a plurality of the sets of synchronization trigger data may be assigned to a specific one of the processing time points, and each of the synchronization trigger data sets includes storage location pointer information for defining a sequence in which the plurality of sets are to be successively read out from memory and processed when the specific processing time point is reached.
According to a third aspect, the overall control and synchronization processing means of such a multimedia playing apparatus comprises means functioning, when the obtained set of synchronization trigger data specifies for the corresponding non base-axis object that playing of the non base-axis object is to be started, to acquire a subject matter data set of the non base-axis object from the subject matter data storage means, to process the subject matter data set into a corresponding formed data set which is in accordance with the playing conditions specified in the scenario, and to supply the formed data set to the second display means.
According to a fourth aspect, such a multimedia playing apparatus further comprises non base-axis object data memory means, and the overall control and synchronization processing means further comprises:
means functioning, during execution of the initial processing operations, to acquire respective sets of subject matter data of each of the non base-axis objects from the subject matter data storage means, to convert each of the sets into respective corresponding sets of formed data which are in accordance with the playing conditions specified in the scenario, and to store the sets of formed data in the non base-axis object data memory means, and
means functioning during the playing progression, when the obtained set of synchronization trigger data specifies for the corresponding non base-axis object that playing of the non base-axis object is to be started, to acquire a formed data set corresponding to the non base-axis object from the non base-axis object data memory means and to supply the formed data set to the second display means.
According to a fifth aspect, such a multimedia playing apparatus further comprises title control command input means, operable by a user for generating a title control input command to specify that a current playing position of the hypermedia title is to be shifted to a new playing position in the playing progression, object presentation status management data memory means, and playing status restoration processing means, and wherein the overall control and synchronization processing means further comprises means functioning during execution of the initial processing operations to obtain the information specifying respective processing time points for starting and ending display of each of the non base-axis objects, to generate corresponding object presentation status management data specifying, for each of respective sequentially numbered segments of the playing progression which are defined between successive pairs of the processing time points, each of the non base-axis objects which is to be played during the segment, and to store the object presentation status management data in the object presentation status management data memory means, wherein the playing status restoration processing means comprises means functioning, in response to designation of a shift to a new playing position by a title control input command, to control the second display means to delete all non base-axis objects which are currently being displayed by the second display means, to obtain the number of a segment which contains the new playing position, to obtain from the object presentation status management data memory means the object presentation status management data corresponding to the segment, to read out from the non base-axis object data memory means respective sets of the formed data of non base-axis objects which are specified for the segment in the object presentation status management data and to supply the non base-axis object formed data to the non base-axis object display means.
According to a sixth aspect, such a multimedia playing apparatus further comprises object control means, and object control command input means operable by a user for generating an object control command to designate for at least one of the non base-axis objects that the non base-axis object is to be displayed in accordance with a user-mediated display condition which is different from a display condition specified for the non base-axis object in the scenario, the overall control and synchronization processing means further comprising means, functioning during execution of the initial processing operations, to modify the formed data set of the non base-axis object to obtain a modified formed data set which is in accordance with the user-mediated display condition, and to store the modified formed data set in the internal data memory means, and
the object control means functioning during playing of the hypermedia title, in response to input of the object control command, to control the second display means to terminate playing of the non base-axis object and to acquire the modified formed data set from the internal data memory means and supply the modified formed data set to the second display means, for thereby playing the non base-axis object in accordance with the user-mediated display condition.
According to a seventh aspect, with such a multimedia playing apparatus the scenario includes information relating to at least one external application program having functions which are not related to the playing of a hypermedia title, the apparatus further comprising external application program activation command input means operable by a user for generating an external application program activation input command, external application program activation means controllable for activating the execution of the external application program, and external application program data memory means, the overall control and synchronization processing means further comprising means functioning during execution of the initial processing operations to generate external application program control data which are necessary for activating the external application program and to store the external application program control data in the external application program data memory means,
the external application program activation means being responsive to generation of the external application program activation input command, during playing of the hypermedia title, for acquiring the external application program control data from the external application program data memory means and supplying the external application program control data to the external application program activation means, and the external application program activation means utilizing the external application program control data to activate the external application program.
According to an eighth aspect, with such a multimedia playing apparatus, the base-axis object subject matter data are cyclically acquired by the base-axis object display means from the subject matter data storage means as successive fixed-size data blocks having a first value of data block size, with a fixed cycle time, and the overall control and synchronization processing means acquires and supplies the non base-axis object subject matter data as successive fixed-size data blocks, with the cycle time, the blocks having a second value of size which is predetermined based on the cycle time and the first value of size, the apparatus further comprising
data block acquisition information memory means having stored therein the cycle time and the second data block size value,
object data acquisition time calculation means functioning during the initial processing to calculate, for each of the non base-axis objects, a duration of acqusition time required to acquire the subject matter data of the each non base-axis object from the subject matter data storage means, and
sequential data acquisition schedule management means for determining, based on the acquisition time durations, respective data acquisition starting time points for sequential acquisition of subject matter data of respective ones of the non base-axis objects to begin, with at least one of the data acquisition starting time points being determined such that at least a part of the subject matter data of a corresponding non base-axis object is specified to be acquired during the playing progression of the hypermedia title,
the overall control and synchronization processing means comprising means functioning during the initial processing to create, in correspondence with each of the data acquisition starting time points which is a positive value, an additional synchronization trigger data set which specifies as processing contents thereof that data acquisition for the corresponding non base-axis object is to commence at the data acquisition starting time point, and to store the additional synchronization trigger data sets in the internal memory means, to acquire from the subject matter data storage means respective subject matter data sets for each of the non base-axis objects corresponding to a data acquisition starting time point which is a negative value, and convert each of the subject matter data sets into a corresponding set of formed data which are in accordance with the playing conditions specified in the scenario, and to store the sets of formed data in the non base-axis object data memory means.
The acquisition of respective subject matter data sets for each of the non base-axis objects corresponding to a data acquisition starting time point which is a negative value may be executed by continuous data acquisition from the subject matter data storage means, or by cyclic acquisition of the subject matter data set from the subject matter data storage means, as sequential data blocks having the second fixed size, with the cycle time.
In addition, the sequential data acquisition schedule management means may comprise means functioning, for each of the non base-axis objects corresponding to a data acquisition starting time point which is a negative number, to divide the subject matter data set of the non base-axis object into an initial processing data acquisition portion and a playing progression data acquisition portion each comprising an integral number of the data blocks of the second size value, and, when the playing progression data acquisition portion has other than zero magnitude, to acquire from the subject matter data storage means the initial processing data acquisition portion and store the portion in the non base-axis object data memory means, and wherein the overall control and synchronization processing means comprises means for creating an additional synchronization trigger data set which specifies that acquisition of the playing progression data acquisition portion from the subject matter data storage means is to be commenced at the starting point of the playing progression, and for storing the additional synchronization trigger data set in the internal memory means.
Moreover it is possible to store in the data block acquisition information memory means a plurality of pairs of values, each pair being a combination of a cycle time value and second data block size value which have been predetermined in accordance with a specific value of bandwidth of a data communication path between the subject matter data storage means and the base-axis object display means. In that way, it becomes possible to permit selection of a cycle time/data block size combination which is appropriate for a particular data communication path.
It can be understood from the above that it is a basic feature of the present invention that, since the starting and termination time points of playing respective monomedia objects (other than the base-axis object) are defined in the scenario in terms of frame numbers of the base-axis object, the synchronization relationships between playing of these monomedia objects and the playing of the base-axis object (and between playing of respective monomedia objects) can be made independent of such factors as variations in playing speed of the base-axis object.
Although the present invention will be described in the following with respect to embodiments in which the non base-axis objects are assumed to be respective still pictures, it should be noted that it would be equally possible for some or all of these to be finite-time monomedia objects, i.e., video clips or audio clips. Whichever is the case, the specific processing time point at which playing of a particular object is to be started, and the time point at which that playing is to be terminated, are designated in respective sets of synchronization trigger data.